Adaptive bit rate ratio control

ABSTRACT

A system for encoding a sequence of frames of a data signal. The system comprises: a first encoding system comprising at least: a first encoder configured to encode the sequence of frames according to a first encoding algorithm; and a first rate control unit configured to control a first bit rate at which the first encoder encodes said sequence of frames; a second encoding system comprising at least: a second encoder configured to encode a second sequence of frames associated with the sequence of frames according to a second encoding algorithm; and a second rate control unit configured to control a second bit rate at which the second encoder encodes said second sequence of frames associated with the sequence of frames.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/844,389, filed Apr. 9, 2020, which is a continuation of U.S. patentapplication Ser. No. 15/779,193, filed May 25, 2018 and granted as U.S.Pat. No. 10,623,745 on Apr. 14, 2020, which is a 371 National StageApplication of PCT/GB2016/053736, filed Nov. 28, 2016, which claimspriority to United Kingdom Application No. 1521013.1, filed on Nov. 27,2015, the disclosures of which are incorporated herein by reference intheir entireties.

TECHNICAL FIELD

The present invention relates to a mechanism to control the bit rateassociated with an encoding system.

BACKGROUND

In nowadays system is important to control the rate at which an encodingsystem generates and transmit encoded data, for example in order toenable a constant bit rate in transmission so that, when a decodingsystem receives an encoded stream to decode, the decoder can receive aconstant bit rate stream (provided the transmission allows a constantbit rate) and decode without having to have too much encoded data todecode or to wait for encoded data before decoding.

The present invention provides a solution to optimise said rate controlusing a specific encoding structure.

SUMMARY

There is provided a system, apparatus, interface and methods accordingto the appended claims.

According to a first aspect of the present invention, there is provideda system for encoding a sequence of frames of a data signal, the systemcomprising a first encoding system comprising at least: a first encoderconfigured to encode the sequence of frames according to a firstencoding algorithm; and a first rate control unit configured to controla first bit rate at which the first encoder encodes said sequence offrames; and a second encoding system comprising at least: a secondencoder configured to encode a second sequence of frames associated withthe sequence of frames according to a second encoding algorithm; and asecond rate control unit configured to control a second bit rate atwhich the second encoder encodes said second sequence of framesassociated with the sequence of frames. The first encoder may be furtherconfigured to: generate a first portion of data comprising datacorresponding to the sequence of frames encoded according to the firstencoding algorithm; conditional on receiving an instruction from thesecond rate control unit to generate a second portion of data comprisingone or more default values for maintaining the first bit rate, generatesaid second portion of data; and generate a first encoded data stream bycombining at least said first and second portions.

In a second aspect there is provided an apparatus for controlling one ormore bit rates associated with corresponding one or more encoders, theapparatus for use within the system above, the apparatus comprising anadaptive rate control unit for adaptively modifying the first bit rateand/or the second bit rate according at least in part on a measure ofcomplexity for the sequence of frames and/or an indication of use ofdefault values by at least one of the two encoding system. The apparatusmay further comprise a receiving unit for receiving an indication that anumber of default values is used or is to be used by at least one of thetwo encoding systems, and wherein the adaptive rate control unit isfurther configured to determine, based at least in part on the receivedindication, to instruct a modification of the second bit rate. Themodification corresponds to a modification of one or more parametersresulting in a higher second bit rate. The apparatus may furthercomprise a transmission unit for sending a signal to the first encodingsystem, wherein the signal comprises instructions for the first encodingsystem to reduce the size of the second portion of data. The apparatusof claim may further comprise a transmission unit for sending a signalto the first encoding system, wherein the signal comprises instructionsfor the first encoding system not to generate any default values for thesecond portion of data. The apparatus may further comprise atransmission unit for sending a signal to the first encoding system,wherein the signal comprises instructions for the first encoding systemto replace one or more of the generated default values with at leastpart of the second encoded data stream. The apparatus may furthercomprise a transmission unit for sending a signal to the first encodingsystem, wherein the signal comprises instructions for the first encodingsystem to insert at least part of the second encoded data stream into atleast a part of the second portion. The apparatus may further comprise atransmission unit for sending a signal to the first encoding system,wherein the signal comprises an indication of a requested second bitrate for the second encoder. The apparatus may further comprise atransmission unit for sending a signal to the first encoding system,wherein the signal comprises an indication of a quantization parameter(QP) for use by the first encoder and/or an indication of a first bitrate for allocation to the first encoder. The apparatus may furthercomprise a receiving unit for obtaining an indication of a second bitrate requested for the second encoder; a decision unit for determiningan optimal bit rate to be allocated for the second encoder, saiddetermination based on a first rate control algorithm; and atransmission unit for sending a signal to the second encoding systemwherein the signal comprises an indication of the second bit rateallocated for the second encoder.

In the system, when used in conjunction with the apparatus, the secondrate control unit may be further configured to control the second bitrate based on an indication of the second bit rate allocated for thesecond encoder. The second encoding system may further comprise ananalysis unit to generate a measure of complexity for the sequence offrames, wherein the complexity is associated with a frame of thesequence of frames and/or with two or more of the frames within thesequence of frames. In the system, when used in conjunction with theapparatus, the first encoder may be further configured to generate anumber of default values for populating said second portion of data,said number being less than a number of default values which would havebeen generated in the absence of said instructions. In the system, whenused in conjunction with the apparatus, the first encoder may be furtherconfigured to inhibit generation of any default values for populatingsaid second portion of data. In the system, when used in conjunctionwith the apparatus, the first encoder may be further configured toinhibiting generation of a number of default values for populating saidsecond portion of data, said number being associated with the size ofthe at least part of the second encoded data stream to be inserted. Inthe system, when used in conjunction with the apparatus, the firstencoder may be further configured to modify the QP used for the encodingprocess based on the signal received and/or to modify the first bit ratebased on the signal received. In the system, the second encoding systemmay further comprise a unit for generating a modified first encoded datastream by modifying the first encoded data stream based at least in parton the second encoded data stream. The system may further comprise amultiplexer for multiplexing the modified first encoded data stream withthe second encoded data stream to generate a multiplexed data stream fortransmission to a decoding system.

In a third aspect there is provided an interface to enablecommunications between the first encoding system and the second encodingsystem, the interface comprising at least means for sending a first datastream from the second encoding system to the first encoding system,said first data stream representing a rendition of the sequence offrames; means for sending a second data stream from the first encodingsystem to the second encoding system, said second data streamcorresponding to a re-constructed version of an encoded rendition of thesequence of frames, said encoded rendition being generated by the firstencoder according to a first encoding algorithm; means for sending afirst signal from the first encoding system to the second encodingsystem, said signal indicating information to be used by the secondencoding system to control the first bit rate and/or the second bitrate; and means for sending a second signal from the second encodingsystem to the first encoding system, said signal indicating informationto be used by the first encoding system to control the first bit rateand/or the second bit rate.

According to a fourth aspect of the present invention, there is provideda method for encoding a sequence of frames of a data signal, the methodcomprising: encoding the sequence of frames using a first encodingsystem; encoding a second sequence of frames associated with thesequence of frames using a second encoding system. The first encodingsystem is adapted to generate a first encoded data stream according to afirst bit rate, the second encoding system is adapted to generate asecond encoded data stream according to a second bit rate. The methodmay further comprise adaptively modifying the first bit rate and/or thesecond bit rate according at least in part on a measure of complexityfor the sequence of frames and/or an indication of use of default valuesby at least one of the two encoding system. The first encoded datastream may comprise a first portion of data comprising datacorresponding to the sequence of frames encoded according to a firstencoding algorithm; and depending on whether it is needed to maintainthe first bit rate in accordance with a first rate control algorithm, asecond portion of data comprising one or more default values.

The method may further comprise measuring a complexity for the sequenceof frames, wherein the complexity is associated with a frame of thesequence of frames and/or with two or more of the frames within thesequence of frames. The method may further comprise sending a signal tothe first encoding system, wherein the signal comprises instructions forthe first encoding system to reduce the size of the second portion ofdata. The method may further comprise generating by the first encodingsystem a number of default values for populating said second portion ofdata, said number being less than a number of default values which wouldhave been generated in the absence of said instructions. The method mayfurther comprise sending a signal to the first encoding system, whereinthe signal comprises instructions for the first encoding system not togenerate any default values for the second portion of data. The methodmay further comprise inhibiting the first encoding system fromgenerating any default values for populating said second portion ofdata. The method may further comprise sending a signal to the firstencoding system, wherein the signal comprises instructions for the firstencoding system to replace one or more of the generated default valueswith at least part of the second encoded data stream. The method mayfurther comprise replacing by the second encoding system one or more ofthe generated default values with at least part of the second encodeddata stream. The method may further comprise sending a signal to thefirst encoding system, wherein the signal comprises instructions for thefirst encoding system to insert at least part of the second encoded datastream into at least a part of the second portion. The method mayfurther comprise inhibiting the first encoding system from generating anumber of default values for populating said second portion of data,said number being associated with the size of the at least part of thesecond encoded data stream to be inserted. The method may furthercomprise generating a modified first encoded data stream by modifyingthe first encoded data stream based at least in part on the secondencoded data stream. The method may further comprise multiplexing themodified first encoded data stream with the second encoded data streamto generate a multiplexed data stream for transmission to a decodingsystem. Encoding the sequence of frames using a first encoding systemmay be performed in accordance with a first encoding algorithm, encodingthe second sequence of frames associated with the sequence of framesusing a second encoding system may be performed in accordance with asecond encoding algorithm. The first encoding algorithm may be a MotionPicture Expert Group (MPEG)-based algorithm, and the second encodingalgorithm may not be an MPEG-based algorithm. The second sequence offrames associated with the sequence of frames is a difference betweenthe sequence of frames and a re-constructed version of the first encodeddata stream.

The second sequence of frames may be up-scaled before or after encodingby the second encoding algorithm or second encoder.

According to another aspect, there is provided a method for encoding asequence of frames of a data signal, the method comprising: receiving afirst sequence of frames encoded using a first encoding system; encodinga second sequence of frames using a second encoding system, wherein thesecond sequence of frames is an up-scaled version of the first sequenceof frames; wherein the first encoding system is adapted to generate afirst encoded data stream according to a first bit rate; wherein thesecond encoding system is adapted to generate a second encoded datastream according to a second bit rate; and wherein the method furthercomprises adaptively modifying at least one of: the first bit rate andthe second bit rate, according at least in part on at least one of: ameasure of complexity for the sequence of frames and an indication ofuse of default values by at least one of the two encoding systems.

The method may include down-sampling the sequence of frames to create adown-sampled sequence of frames for encoding by the first encodingsystem. The method may further include providing the first encodingsystem with the down-sampled sequence of frames, wherein saiddown-sampled sequence of frames is to be processed by the first encodingsystem to generate the first sequence of frames.

According to another aspect, there is provided a system for encoding asequence of frames of a data signal, the system comprising: a firstencoding system and a second encoding system. The first encoding systemcomprising: a first encoder configured to encode a first sequence offrames according to a first encoding algorithm, wherein the firstsequence of frames is a down-scaled version of the sequence of frames;and a first rate control unit configured to control a first bit rate atwhich the first encoder encodes said first sequence of frames. Thesecond encoding system comprising: a second encoder configured to encodea second sequence of frames according to a second encoding algorithm,wherein the second sequence of frames is an up-scaled version of thefirst sequence of frames; and a second rate control unit configured tocontrol a second bit rate at which the second encoder encodes saidsecond sequence of frames associated with the sequence of frames.

Further features and advantages will become apparent from the followingdescription of embodiments, given by way of example only, which is madewith reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic block diagram of an example of an encodersystem in accordance with an embodiment of the present invention;

FIG. 2 shows a schematic block diagram of an example of an encodersystem in accordance with an embodiment of the present invention;

FIG. 3 shows a schematic block diagram of an example of an encodersystem in accordance with an embodiment of the present invention; and

FIG. 4 shows a schematic block diagram of an example of an encodersystem in accordance with an embodiment of the present invention

DETAILED DESCRIPTION

Referring to FIG. 1, there is shown a schematic block diagram of anexample of an encoder system in accordance with an embodiment of thepresent invention. In an example, second encoding system receives aconfiguration stream which is then used by configuration manager to sendcontrol signals to configure the various elements of the second encodingsystem, namely down scaler, up scaler, second encoder and rate controlunit. The configuration stream includes parameters which relate tospecific configuration settings of the second encoding system. Forexample, these parameters include parameters relating to down-scalingand/or up-scaling operations (e.g., type of filters, filter parameters),bit rate parameters, information about the size of the data stream(e.g., resolution, frame rate, etc.) and other relevant parameters.Second encoding system also receives a data stream which needs to beencoded. The data stream could correspond to a stream of raw data from asource. For simplicity and convenience, in the description we willassume that data stream is a video data stream generated by a videorecording device at a certain level of quality. For example, the encodeddata could include any other type of data, including a sound signal,multichannel sound signal, picture, two-dimensional image, multi-viewvideo signal, 3D video signal, volumetric signal, volumetric videosignal, medical imaging signal or a signal with more than fourdimensions, and all the apparatuses, systems and methods describedherein should apply mutatis mutandis to those other types of datasignals. Data stream is then provided to a down scaler which produces adata stream at a lower level of quality, i.e. reduced resolution datastream. For example, data stream could be a video signal at 1920×1080 p,whereas the reduced resolution data stream could be the same videosignal reduced to 960×540 p. The reduced resolution data stream istransmitted via API to a first encoding system which includes a firstencoder and a rate control unit. Reduced resolution data stream is thenencoded by first encoder using a first encoding algorithm (e.g., astandard-based MPEG encoding algorithm such as H.264). The first encoderoutputs two streams, a first stream corresponding to an encoded versionof the reduced resolution data stream, namely encoded data stream, and asecond stream corresponding to a decoded version of the encoded datastream, namely decoded data stream. The latter is then provided via APIto up scaler which produces an up-scaled data stream. Importantly, theresolution of data stream and that of up-scaled data stream areequivalent.

At this point, a difference is taken between data stream and up-scaleddata stream to produce a difference data stream. The difference datastream is encoded using a second encoder. Said second encoder typicallyuses a second encoding algorithm. Such second encoding algorithm takesdifference data stream, applies a specific set of transformationmatrices and encodes the resulting transformed data stream using anentropy encoder to produce an encoded reconstruction data stream. Thesecond encoding algorithm may also include a quantization process beforeuse of the entropy encoder. A full discussion on this second encodingalgorithm, how it works, and what type of matrices are used is describedin International patent application Pub. No. WO 2013/171173 which isincorporated herein by reference. Said reconstruction data stream canalso include information on how to reconstruct a rendition of datastream starting from a decoded version of encoded data stream.

Reconstruction data stream can be provided to a supplemental enhancementinformation (SEI) unit to produce data stream. Typically, the SEI unitis adapted to combine into a single elementary stream two or more SEIencapsulated streams, in the present case reconstruction data stream andencoded data stream. The encapsulation may be made by the SEI unitdirectly, or it could be fed to the SEI unit after encapsulation. Thefunctioning of the SEI is based on standards such as ITU-T H.264.Alternatively (or in addition), encoded data stream and reconstructiondata stream are multiplexed together by transport stream multiplexer toproduce a multiplexed transport data stream. Alternatively, or inaddition (but not shown in the Figure) single elementary stream andreconstruction data stream can be multiplexed together by transportstream multiplexer (or another transport stream multiplexer) to producea multiplexed transport data stream.

The reconstruction data stream provides some significant advantages,some of which are explained here. A first advantage is to allow for thereproduction after decoding at the decoder side of a first qualityand/or resolution video stream (e.g., High Definition, HD) starting froma second quality and/or resolution video stream (e.g., StandardDefinition, SD) which would be otherwise provided by decoding only theencoded data stream produced by the first encoding system, the firstquality and/or resolution being higher than the second quality and/orresolution. The overall bit rate used by the combination of thereconstruction data stream and the encoded data stream is lower than thebit rate which would be required by the first encoder to produce anencoded data stream which, when decoded and played at the decoder side,would result in a video stream of quality and/or resolution comparableto that of the first quality and/or resolution. Another advantage itthat is allows back-compatibility with existing decoding system and/orcompatibility with multiple device types, whereby one existing decodingsystem could decode based only on the encoded data stream and anotherexisting device system could decode based on the on both the encodeddata stream and the reconstruction data stream.

The multiplexed data stream and/or the single elementary stream is thensent over a transmission channel (e.g., over the air, cable, etc.) to adecoding system (not shown) which would then use the combination ofencoded data stream and reconstruction data stream to reconstruct arendition of the original data stream. In particular, encoded datastream is provided to a first decoder which decodes encoded data streamusing a decoding mechanism which corresponds to the encoding mechanismused by the first encoder (e.g., a standard-based MPEG decodingalgorithm such as H.264). The stream so decoded is then used as a “base”layer by a second decoder which combines it with reconstruction datastream to reconstruct a rendition of the original data stream. Thissecond decoder uses a decoding mechanism which corresponds to theencoding mechanism used by the second encoder. More details on theencoding and decoding mechanisms, as well as on the overall mechanismsdescribed above can be found in International patent application Pub.No. WO 2014/170819 which is incorporated herein by reference.

The ratio between the bit rate allocated and/or associated with thefirst data stream (e.g., encoded data stream) encoded with firstencoding system and the bit rate allocated and/or associated with thesecond data stream (e.g., reconstruction data stream) encoded with thesecond data system is an important factor in ensuring that the qualityof the reconstructed rendition of the original data stream is optimisedwhilst the bit rate of the multiplexed data stream is kept undercontrol. A typical range of ratio values is between 60:40 and 90:10,where the first number indicates the percentage of bit rate allocatedand/or associated with the first data stream and the second numberindicates the percentage of bit rate allocated and/or associated withthe second data stream. A potential ratio value is 70:30.

In the specific example of FIG. 1, each encoder may be provided with acorresponding rate control unit which is used to control the data rateproduced by the respective encoder during the encoding process. Inparticular, rate control unit controls the data rate of second encoder,with rate control unit controlling the data rate of first encoder.

Importantly, rate control unit typically interacts with first encoder toensure that encoded data stream is such that a certain reference bitrate (e.g., a Constant Bit Rate CBR) is maintained. Rate control unitsare well known in the art. An example of how they work is described in“A Generalized Hypothetical Reference Decoder for H.264/AVC”, IEEETransactions On Circuits and Systems for Video Technology, Vol. 13, No.7, Jul. 2003 by J. Ribas-Corbera et. Al., whose contents areincorporated herein by reference. In particular, this paper describesthe concept/strategy of the leaky bucket. This rate control techniquelooks at a rate of data generated by an encoder and a rate of encodeddata to be sent by the encoder at a specific rate (e.g., CBR). Since therate of data generated by the encoder is typically variable over time,the amount of encoded data to be sent by the encoder may fall below athreshold as the encoded data are sent at said specific rate. Forexample, if the specific rate is a CBR of 3 Mbits/s, and the encodergenerates for the first two seconds 6 Mbits and for the next fourseconds 10 Mbits, that implies that for the first two seconds the bucketshould remain at the same level it was before generation of those 6Mbits of encoded data (since the data are transmitted at a constant bitrate of 3 Mbit/s) but then for the next four seconds the level ofencoded data in the bucket would reduce by 2 Mbits (as there would be 12Mbits of encoded data transmitted with “only” 10 Mbits of data generatedover the same period). If the level of encoded data available fortransmission falls below a certain threshold, the rate control unitwould instruct the encoder to “compensate” for this shortfall ofavailable encoded bits by generating more encoded bits. This, forexample, could be done by changing the encoding parameters such as QP inorder for the encoder to use more bits for encoding. If the encoder isnot capable of generating additional encoding bits to move the level ofavailable encoded bits above the threshold, then the rate control unitwould enable generation of default bits (e.g., zeros) to “fill” thebucket of available data, thus ensuring that the constant bit rate oftransmission is maintained. Conversely, there could also be a higherthreshold that should be monitored to avoid that there are too manyavailable encoded bits compared to the rate at which they can betransmitted, as in that case it would create a problem at the decoderside where frames to be decoded could be dropped as a consequence. Forexample, if the level of encoded data available for transmission goesabove a certain threshold, the rate control unit would instruct theencoder to generate less encoded bits by, for example, changing the QPso that a less finer quantization is performed.

During experimental tests performed on video signals (e.g., a sequenceof video frames) it was observed that in scenes with high spatialcomplexity and low temporal complexity the first encoder required a lowbit rate, and therefore only a small percentage of the available bitrate should be given to the first encoder allowing the second encoder touse a higher bit rate by reusing all or part of the remaining bit rate.On the other hand, in scenes with low spatial complexity but hightemporal complexity it is best to give most of the available bit rate tothe first encoder. In general, spatial complexity is inverselyproportional to spatial correlation present within a single frame. Inother words, high spatial complexity means that the elements in thescenes are less correlated (e.g., a scene where there are a large numberof details of non-repetitive nature, for example a scene of a crowd at afootball stadium) whereas low spatial complexity means that the elementsin the scene are more correlated. Additionally, temporal complexity isinversely proportional to temporal correlation between frames. In otherwords, high temporal complexity means that consecutive frames are lesscorrelated (e.g., they are significantly different, for example becausemany elements change positions and/or shapes), whereas low temporalcomplexity means that consecutive frames are more correlated.

One reason for this behaviour is that the first encoder may be optimisedfor utilising the underlying temporal correlation in a sequence offrames in an efficient manner which typically results in a bettercompression rate. For example, in the case MPEG-based encodingalgorithms, a typical sequence of frames is encoded by having Group ofPictures (GOPs) in which an initial I-frame (e.g., a frame which can bedecoded only using data encoded for that specific frame) is followed bya series of P-frames (i.e. frames which require data encoded fromprevious frames for decoding, but because of this allow for highercompression rates than I-frames) and/or B-frames (i.e., frames whichrequire data encoded from previous and subsequent frames for decoding,but because of this allow for higher compression rates than I-frames andP-frames). The higher successive frames are correlated, the moreP-frames and B-frames can be used, and therefore the encoder willrequire lower bit rates. On the other hand, the second encoder isusually one that maximizes spatial correlation, and therefore when thespatial correlation is higher, using a higher bit rate for the secondencoder will allow to increase the quality of the reconstructedrendition of the original data stream by using more bits for the seconddata stream.

A further observation is that, when low bit rates are available and/orused for transmission, in order to have a sufficiently good level ofquality for the reconstructed rendition of the original data stream inthe event of temporally complex scenes, a big proportion of theavailable and/or used bit rate should be given to the first encoder (forexample, equal or more than 85%). However, that would mean setting aratio, in the current example of 85:15, which in turn implies that whenfor example there is a scene change with the new scene being very sharp(i.e., high spatial complexity) with low temporal complexity, it islikely that 15% of bit rate is not enough for the second encoder toprovide a sufficient amount of reconstruction data over thereconstruction data stream for obtaining a sufficiently good level ofquality for the reconstructed rendition of the original data stream. Atthe same time, because of the low temporal complexity the first encoderwould only be using a small portion of bits it is allowed and generatefiller in order not to underflow the decoder buffer. This fillercorrespond to “waste” bits which are only inserted in the bit stream inorder to keep a constant bit rate but without any further benefit forthe encoded sequence.

Accordingly, a possible solution would be to prevent the first encoderfrom putting the filler in the bit stream and instead adding bits fromthe reconstruction data stream so as to decrease the ratio and giving abigger percentage of the available bit rate to the reconstruction datastream.

In an exemplary embodiment as shown in FIG. 2, this is achieved by thefirst encoding system sending to the second encoding system via the API140 additional information. This additional information may be generatedby rate control unit based on a specific rate control mechanismassociated with first encoder. Said additional information may compriseinformation about the size of the data stream encoded by the firstencoder and the size of the filler that the first encoder has added oris planning to add into said encoded data stream. In one example, sizeof the data stream encoded by the first encoder can be expressed as thebyte-size of the data stream encoded by the first encoder, whereas thesize of the filler can be expressed as the byte-size that the firstencoder desires to use for the base filler. This additional informationis provided to the rate control unit in order to enable it to controlthe bit rate of the reconstruction data stream generated by the secondencoder.

In particular, rate control unit, based on information about thecombined transmission of the encoded data (e.g., encoded data stream andreconstruction data stream), can determine that the portion of theencoded data stream which is or is intended to be used for filler couldbe instead used for adding more data for the reconstruction data stream.For example, rate control unit can instruct second encoder to encodeusing more bits, thus increasing the bit rate of the reconstruction datastream, and consequently allowing for an improvement of the data streamonce decoded. One mechanism for the described determination is the useof a “joint” leaky bucket which is a modification of the leaky bucketdescribed above. In particular, the “joint” leaky bucket is used to keeptrack of the combined encoded data available for transmission (i.e.,both the available encoded data generated by the second encoder 116 andthe available encoded data generated by the first encoder) and maintaina specific combined transmission rate (e.g., a CBR). As in the mechanismdescribed above, the rate control unit using said “joint” leaky bucketwould control the second bit rate of the second encoder in order toensure that the combined transmission rate is maintained. In particular,if detected that filler is or is to be inserted by the first encodingsystem, it would instruct second encoder to generate more bits (e.g., byhaving a finer granularity in the quantization process) in order to usethem in place of all or part of the filler. Then, the rate control unitwould send a signal to instruct the first encoding system via the API toavoid generating such filler. In addition, the rate control unit willalso use the “joint” leaky bucket to determine whether to increase thenumber of bits generated by the second encoder when the level ofcombined available encoded data falls below a threshold. This is becausein that case the second encoder could “compensate” for the shortfall bygenerating more bits (e.g., by having a finer granularity in thequantization process) thereby increasing the quality of thereconstruction data stream. If the second encoder is unable to furtherincrease the bits used for encoding and the level of combined availableencoded data is still below a threshold, then the rate unit may enablegeneration of filler by the second encoding system. In this way, thecombined stream timing and buffer management does not change but animprovement in the quality of the video played at the decoder isachieved.

Alternatively (or in addition), rate control unit may decide that, basedon the temporal and/or spatial complexity of the original data stream,it would be beneficial to dedicate more bits to the second encoder so asto generate more bits for the reconstruction data stream. For example,as described above, if the scene has high spatial complexity and lowtemporal complexity, then rate control unit may decide to dedicate morebits to the second encoder. In such case, upon verifying based on theadditional information that the first encoding system intends to usesome filler, rate control unit may send a signal to instruct the firstencoding system via the API to avoid generating such filler. This signalmay be sent to rate control unit which in turn may instruct the firstencoder not to generate this filler. Alternatively, rate control unitmay send a signal to indicate to the first encoding system via the APIthat a certain number of bits are required for the reconstruction datastream and should be “reserved” by the first encoding system. As aconsequence, the first encoding system should only generate filler (ifany) for up to the difference between the size of the filler that thefirst encoder has added or is planning to add into said encoded datastream and the number of bits required for the reconstruction datastream.

The temporal and/or spatial complexity may be measured in variousmanners. One possible way is by measuring the entropy associated with aframe and/or a sequence of frames within the original data stream. Theentropy is defined as the expected value (e.g., average) of theinformation contained in a data set. For example, in the case of aframe, it can be seen as the expected value of the information containedin that frame, so that a frame with a high content of information (e.g.,a frame showing a crowd in a stadium) will have a high entropy. In thecase of a sequence of frames, it could be seen as the expected value ofthe information across frames (e.g., a sequence of frames with fastmoving objects). The measured entropy is compared against a threshold,and if above said threshold the corresponding complexity is determinedas being high. Conversely, if below said threshold, the correspondingcomplexity is determined as being low. Of course, multiple thresholdscould be used, for example two thresholds, a higher one above whichcomplexity is determined to be high, and a lower one below whichcomplexity is determined to be low. The measured entropy could refer toa spatial entropy (e.g., the entropy within a frame), a temporal entropy(e.g., the entropy between frames), or a combination of the two. Thethresholds could then be adapted accordingly to account for the exacttype of corresponding complexity. For example, in the case of entropyassociated both with a temporal and a spatial complexity, there could beeither a combined set of thresholds (i.e., thresholds that take intoaccount both spatial and temporal complexity—this would be the case, forexample, of a measure of combined entropy which is both a function ofspatial entropy and temporal entropy) or two pairs of sets ofthresholds, one set per dimension of entropy (i.e., one set for spatialentropy, the other for temporal entropy). The latter would be the caseof a measure of entropy which includes two separate entropymeasurements, one for spatial entropy and another for temporal entropy.

In a further embodiment, signal may also include a minimum value for theQuantization Parameter (QP) which is to be used by the first encoder.The QP is used for quantizing data when performing lossy compression.This in turn would enable to control the size of the data stream encodedby the first encoder and of the size of the filler so that the rate ofbit rate of the reconstruction data stream can be managed in a mannerthat ensures sufficiently good level of quality for the reconstructedrendition of the original data stream. For example, this mechanism couldallow to decrease the size of the filler and therefore enable more bitrate for the reconstruction data stream, thus increasing the quality ofthe video played at the decoder side.

In a further embodiment, it was observed that for high complexity scenes(e.g., both temporally and spatially) both the first encoder and thesecond encoder would have problems in encoding in an efficient mannerusing a lossy compression mechanism, and therefore the resulting encodedstream is likely to produce artefacts when decoded. From a visual andperceptual perspective (e.g., from the perspective of a person viewingthe stream once decoded) temporal and low resolution artefacts caused bythe first encoder are more noticeable and affect the visual perceptionof the decoded stream, thus resulting in a perception of an overall lowquality of the decoded stream and an unsatisfactory visual experience.Such artefacts are, for example, “blocky” scenes (i.e., scenes whereblocks of the frames are either disappearing or whose boundaries areevident when viewing the frame) or black spots in the frame. Thosetemporal and low resolution artefacts negatively affect the visualperception of the decoded stream to a greater extent than if the streamonce decoded was to lack in high resolution details (e.g., due to thelack of bit rate given to second encoder). Note that the latter areusually the details added when decoding the reconstruction data streamproduced by the second encoding system. Accordingly, it was concludedthat it may be better to use most of the combined bit rate for the firstencoder (thus increasing the first bit rate), so that a lower QP couldbe used instead and therefore more bits were given to the encodedstream. Accordingly, when it is determined that the scene is highlycomplex, a signal can be sent from second encoding system to firstencoding system via the API in order, for example, to lower the QP usedby the first encoder. For example, this signal can be sent as part ofsignal.

In an exemplary embodiment as shown in FIG. 3, rate control unit may usesome statistical analysis (e.g., look-ahead statistics) to determine thebest ratio in order to ensure a sufficiently good level of quality forthe reconstructed rendition of the original data stream. In thisembodiment, rate control unit would simply send an instruction via APIto rate control unit indicating the bit rate available for the secondencoder. In turn, rate control unit would then use this indication tocontrol the bit rate of the second encoder. In addition, the secondencoding system may send a signal via API to the first encoding system,said signal indicating a request for a bit rate to be allocated to thesecond encoder and/or a measure of complexity of the scene. For example,the second encoding system may determine an optimal bit rate to beallocated to the second encoder in order to ensure a sufficiently goodlevel of quality for the reconstructed rendition of the original datastream. In addition, the second encoding system may determine an optimalbit rate to be allocated to the second encoder based on an estimatedcomplexity for a scene (spatial and/or temporal complexity). The optimalbit rate could be determine as discussed above. The second encodingsystem may also estimate the complexity for a scene and generate ameasure of complexity based on said estimate. The measure of complexitycan be used by the first encoding system (e.g., by the rate controlunit) to control the bit rate for the first encoder and/or fordetermining bit rate to be made available for the second encoder.

FIG. 4 shows a variation which could be made to any of the otherFigures, in particular in relation to what happens after decoded datastream is received by the second encoding system. Rather than beingup-scaled by up-scaler, a difference is taken between reduced resolutiondata stream and decoded data stream. Said differential data streamrepresents a correction that would need to be made to decoded datastream in order to obtain reduced resolution data stream. Thisdifferential data stream is encoded by second encoder to generate anencoded correction data stream (not shown) by using the second encodingalgorithm as described above. Said corrected encoded data stream couldthen be added to the final output of the second encoder in addition (ortogether with) reconstruction data stream, for example to form a pair ofdata streams to reconstruct at the decoder. Said encoded data stream(not shown) is first decoded to generate a decoded corrected datastream, which is then summed to decoded data stream to generate acorrected encoded data stream. Said corrected decoded data streamrepresents a corrected version of the decoded data stream. Thiscorrected decoded data stream is then up-sampled to generate up-scaleddata stream, and then processed as described above in relation toFIG. 1. The additional processing described above with reference to FIG.4 allows for having the second encoding system generating two datastreams for combination with the encoded data stream, a first datastream (the not shown encoded correction data stream) to “correct” theencoded data stream and bring it, when reconstructed at the decoder, tothe level of quality of reduced resolution data stream, the second datastream (reconstruction data stream) to “enhance” the encoded data streamafter correction and bring it, when reconstructed at the decoder, to thelevel of quality of the data stream. In other words, the decoder caneither simply “correct” the encoded data stream, or “correct” and“enhance” to a better resolution the encoded stream.

Although at least some aspects of the examples described herein withreference to the drawings comprise computer processes performed inprocessing systems or processors, examples described herein also extendto computer programs, for example computer programs on or in a carrier,adapted for putting the examples into practice. The carrier may be anyentity or device capable of carrying the program.

The use of a modular structure such as the one depicted in any of theFigures provides also an advantage from an implementation andintegration point of view, enabling a simple integration into legacysystems as well as compatibility with legacy systems. By way of example,the second encoding system could be embodied as a plug-in (includinglibraries and/or source code) to an existing firmware and/or softwarewhich already embodies legacy first encoding system (for example onethat is already installed in legacy encoder systems). The first encodingsystem and the second encoding system may be embodied as a singlesystem, or as two separate systems. In addition, the API could need tobe included, and the necessary modifications to the first encodingsystem would need to be made based on the specific embodiment required.The first encoding system may provide to the second encoding system theencoded data stream or the decoded version of the encoded data stream.

It is to be understood that any feature described in relation to any oneembodiment may be used alone, or in combination with other featuresdescribed, and may also be used in combination with at least one featureof any other of the embodiments, or any combination of any other of theembodiments. Furthermore, equivalents and modifications not describedabove may also be employed without departing from the scope of theinvention, which is defined in the accompanying claims.

1. (canceled)
 2. A system for encoding a sequence of frames of an inputdata signal, the system comprising: a second encoding system comprisingat least; a down-scaler to down scale the sequence of frames of theinput data signal to generate a first sequence of frames; an interfaceto enable communications between the second encoding system and a firstencoding system, wherein the first encoding system comprises at least afirst encoder configured to encode the first sequence of framesaccording to a first encoding algorithm and a first rate control unit tocontrol a first bit rate at which the first encoder encodes the firstsequence of frames, wherein the first encoding system outputs a firststream corresponding to an encoded version of the first sequence offrames, wherein the second encoding system further comprises at least:an up-scaler to up scale a second stream corresponding to a decodedversion of the first stream to generate a second sequence of framesassociated with the first sequence of frames; a second encoderconfigured to encode data derived from the second sequence of framesaccording to a second encoding algorithm to output a third stream; asecond rate control unit to control a second bit rate at which thesecond encoder encodes the data derived from the second sequence offrames; and a configuration manager configured to receive configurationparameters and to control the first encoding system and the secondencoding system, wherein the configuration manager is configured tocommunicate with the first rate control unit and the second rate controlunit to maintain a specified combined transmission rate for the firstand third streams.
 3. The system of claim 2, wherein the second encodingsystem is configured to encode a first difference between the sequenceof frames of the input data signal and the second sequence of frames. 4.The system of claim 3, wherein the second encoding system is configuredto compute a second difference between the first sequence of frames andthe second stream and to further encode the second difference togenerate an encoded correction data stream, the encoded correction datastream forming part of the third stream output by the second encoder. 5.The system of claim 4, wherein the second encoding system is configuredto decode the encoded correction data stream to generate a decodedcorrection data stream, the decoded correction data stream being summedwith the second stream prior to up-scaling by the up-scaler.
 6. Thesystem of claim 2, wherein the first encoding system comprises a legacyfirst encoding system and the system comprises a plug-in for the legacyfirst encoding system.
 7. The system of claim 2, wherein theconfiguration manager is configured to measure a temporal and/or spatialcomplexity of frames of the input data signal and to dynamically controlthe first bit rate and second bit rates based on the measured temporaland/or spatial complexity.
 8. The system of claim 7, wherein thetemporal and/or spatial complexity is based on one or more measures ofentropy for one or more frames of the input data signal.
 9. The systemof claim 2, wherein a minimum value for a Quantization Parameter (QP) tobe used by the first encoder is communicated from the second encodingsystem.
 10. The system of claim 9, wherein the value of the QuantizationParameter (QP) is lowered for more complex frames and raised for lesscomplex frames.
 11. The system of claim 2, wherein the first ratecontrol unit is configured to send an instruction via the interface tothe second rate control unit indicating a bit rate available for thesecond encoder.
 12. The system of claim 11, wherein the second encodingsystem is configured to send a signal via the interface to the firstencoding system indicating one or more of a request for a bit rate to beallocated to the second encoder and a measure of complexity for a frame.13. The system of claim 2, wherein the configuration manager isconfigured to control a size of filler data inserted into the firststream, wherein the third stream is used in place of the filler datawithin a combined stream comprising the first and third streams.
 14. Thesystem of claim 2, wherein the configuration manager is configured todynamically control a ratio of the first bit rate to the second bitrate.
 15. A method for encoding a sequence of frames of an input datasignal, the method comprising: downscaling the sequence of frames of theinput data signal to generate a first sequence of frames; receiving adecoding of an encoding of the first sequence of frames, the encodingbeing generated using a first encoding system; upscaling data derivedfrom said decoding to generate a second sequence of frames associatedwith the first sequence of frames; and encoding data derived from thesecond sequence of frames using a second encoding system, wherein thefirst encoding system is adapted to generate a first encoded data streamaccording to a first bit rate, wherein the second encoding system isadapted to generate a second encoded data stream according to a secondbit rate, and wherein the method further comprises: receivingconfiguration parameters to control the first encoding system and thesecond encoding system; and communicating with the first encoding systemand the second encoding system to maintain a specified combinedtransmission rate for the first and second encoded data streams.
 16. Themethod of claim 15, comprising: determining a first difference betweenthe sequence of frames of the input data signal and the second sequenceof frames; encoding the first difference as part of the second encodeddata stream; determining a second difference between the first sequenceof frames and the decoding of the encoding of the first sequence offrames; and encoding the second difference as part of the second encodeddata stream.
 17. The method of claim 16, wherein a decoding of thesecond difference is added to the decoding of the encoding of the firstsequence of frames prior to upscaling.
 18. The method of claim 15,further comprising: measuring a temporal and/or spatial complexity forframes of the input data signal; and dynamically controlling the firstbit rate and second bit rates based on the measured temporal and/orspatial complexity.
 19. The method of claim 15, further comprising:communicating a minimum value for a Quantization Parameter (QP) to beused by the first encoding system from the second encoding system,wherein the value of the Quantization Parameter (QP) is lowered for morecomplex frames and raised for less complex frames.
 20. The method ofclaim 15, further comprising: dynamically controlling a ratio of thefirst bit rate to the second bit rate.
 21. A non-transitory computerreadable medium comprising instructions that, when executed by aprocessor, cause the processor to: receive configuration parameters tocontrol a first encoding system and a second encoding system; downscalea sequence of frames of an input data signal to generate a firstsequence of frames; receive a decoding of an encoding of the firstsequence of frames, the encoding being generated using the firstencoding system; upscale data derived from said decoding to generate asecond sequence of frames associated with the first sequence of frames;and encode data derived from the second sequence of frames using thesecond encoding system, wherein the first encoding system is adapted togenerate a first encoded data stream according to a first bit rate,wherein the second encoding system is adapted to generate a secondencoded data stream according to a second bit rate, and wherein thefirst bit rate of the first encoding system and the second bit rate ofthe second encoding system are controlled based on the configurationparameters to maintain a specified combined transmission rate for thefirst and second encoded data streams.